A portion of the disclosure of this patent document contains command formats and other computer language listings all of which are subject to copyright protection. The copyright owner, EMC Corporation, has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
1. Field of the Invention
The present invention relates generally to a file server employing a plurality of processing units. More particularly, the invention relates to failover services for resuming interrupted operations of a failed processing unit with little or no client involvement.
2. Background Art
Transactional semantics are typically employed between a host computer or client and a data storage system or file server to permit recovery from a failed processor in the data storage system or file server. The host computer or client sends to the data storage system or file server a command or command chain defining a transaction for which the data storage system or file server must acknowledge completion before committing results of any following commands.
In an environment employing commands in the IBM Corporation count-key-data (CKD) format, for example, all of the commands for a single input/operation for a single logical volume are included in a single channel command word (CCW) chain. The data storage system acknowledges completion of each write command by returning a channel end (CE) and device end (DE) to the host computer. The results of all channel command words of this single input/output operation are to be committed before commitment of the results of any following CCW""s. Once the host processor sends the entire chain to the data storage system, it need not poll for a response; instead, the host typically continues with other operations, and is interrupted when the channel adapter responds with a device end (DE) signal indicating that the results of the last CCW in the chain has been committed.
In an open systems environment, a data storage system typically handles each input/output command as a separate transaction and acknowledges completion of each input/output command. If a problem occurs, the data storage system containing the primary returns a xe2x80x9cunit checkxe2x80x9d with appropriate sense bytes to the host. This causes the host to retry the input/output operation.
By employing transactional semantics, a failure of a redundant processor in the data storage system or file server will not usually disrupt the operation of the host computer or client any more than a simple failure of the data link between the data storage system or file server and the host computer or client. Upon failing to receive an acknowledgement of completion of a transaction, the host computer or client re-sends the transaction. If the data storage system or file server continues to fail to acknowledge completion of the transaction, the host computer or client may re-send the transaction over an alternative data link to the data storage system or file server.
The use of transactional semantics and the re-try of unacknowledged transactions is a good technique for contending with processor failures in a data storage system or file server in which the transactions are primarily read and write operations. However, network-attached file servers, and video file servers in particular, perform data streaming operations for which the re-try of unacknowledged transactions has some undesirable consequences. A data streaming operation requires exclusive use of certain valuable resources, such as buffer memory, a dedicated port in the file server, and a dedicated network data link. Therefore, the file server should detect processor failure without reliance on the client in order to free-up the dedicated resources as soon as possible. Moreover, a data streaming operation that directs data to a network destination other than the client may also involve other clients or consumers having a special interest in minimizing delay or disruption in the streaming of the data. For example, in a video file server application, the data may be viewed in real time by an ultimate consumer, and any delay in transmission in excess of the consumer""s buffering capabilities will interfere with the consumer""s viewing pleasure.
The present application is directed to improvements to the failover services disclosed in Duso et al., U.S. application Ser. No. 08/851,507 filed May 5, 1997, issued as U.S. Pat. No. 5,987,621 on Nov. 16, 1999, incorporated herein by reference. In particular, the present invention is directed to a file server that includes a plurality of stream server computers linking data storage to a data network for transfer of data streams between the data storage and the data network, and at least two controller servers for controlling the stream server computers. The controller servers are programmed so that one of the controller servers becomes active in controlling the stream server computers, and another of the controller servers becomes inactive in controlling the stream server computers. A controller server failover mechanism is provided for recovering from a failure of the active controller server. The inactive controller server is programmed to respond automatically to a failure of the active controller server by becomming active in controlling the stream server computers.
In accordance with one aspect of the invention, each of the controller servers has a respective flag for indicating whether or not the controller server is active or inactive in controlling the stream server computers. The flags, for example, are stored in local memory of the stream servers, or they are stored in the data storage. In any case, each controller server computer can set or reset its own flag and can read the flag of the other controller server computer. Each controller server computer is programmed so that upon booting, it will read the flag of the other controller server computer, and if the flag of the other controller server computer indicates that the other controller server computer is active in controlling the stream servers, then the controller server computer becomes inactive in controlling the steam server computers. Otherwise, if the flag of the other controller server computer indicates that the other controller server computer is inactive, then the controller server computer assumes active or inactive status based on a predetermined arbitration method. For example, one of the controller servers could be programmed to assume inactive status in this situation, and the other of the controller servers could be programmed to assume active status in this situation. Preferably, however, the arbitration method is based on a numerical slot number of a slot in a rack in which the controller servers are placed. For example, when installed into the rack, each controller server can read the respective slot number which has been wired into a mating connector of the rack. This eliminates any need for different hardware or programming for each of the controller servers.
In accordance with another aspect of the invention, the active controller server monitors the inactive controller server to ensure that the inactive controller server is kept in a state of readiness to assume active status. If the inactive controller server is found not be in a state of readiness to assume active status, the active controller server reports the error condition to service personnel. For example, each controller server has a dial-up modem that can automatically place a call to a customer service center and transmit diagnostic information related to the error condition. The inactive controller server, for example, normally transmits a signal periodically to the active controller server, and the active controller server senses an error condition upon failing periodically to receive the signal. The active controller server may also reboot the inactive controller server in an attempt to put the inactive controller server in a state of readiness. Moreover, the inactive controller server can be running its own diagnostic program routines, and upon detecting an error condition, the inactive controller server can report the error condition to the active controller server. The active controller server then reports the error condition to service personnel.
In accordance with another aspect, the active controller server runs its own diagnostic routines, and upon detecting a failure, it reports the failure to the inactive controller. The inactive controller then begins to assume active status and reboots the active controller server, which then assumes inactive status upon determining that the inactive controller server has begun to assume active status.